Back To Blog

Data Factory. Advantages & Implementation

Data Factory is an architectural concept and a set of approaches to working with data. It is important to understand that this is not a “boxed” solution, where you can buy, install, configure integration with the existing systems, and then successfully operate. We are talking about methodologies, in order to understand how they work and how to correctly apply them.

Big data is not static, it is constantly changing: new sources appear, some data quickly lose relevance. It is not possible to maintain all this history within a single corporate data warehouse. Therefore, the Data Factory is both corporate data warehouses, and data lakes, clouds, the Internet of Things, and streaming data all combined.

Data factories have evolved from data warehouses, therefore, conceptually, three large layers of technological implementation can be distinguished:

1. integration layer

2. storage layer

3. service layer

Also, a data management system, DevOps and DataOps services should be added to the data factory, which allows to correctly configure and manage the life cycle of implementing various data flows and handling them.

If we consider the conceptual architecture in more detail and study the integration layer at the technological level, we can note the classic ETL / ELT tools. It is worth noting that now there is a trend: instead of enterprise solutions popular a few years ago from large vendors, open-source tools such as NiFi, Airflow, with the ability to write integrations either at the database level or at the level of additional scripts, come to the fore in Python, which are comparable to enterprise solutions. They have a higher entry threshold, but this is offset by the flexibility and cost of the software itself, which is often either very low or free. Speaking of integration, we should not forget about streaming platforms, message-brokers, such as RabbitMQ, Kafka, which allow you to transfer large data streams in real time, while still making relatively simple transformations with them.

The next layer is the storage layer. This component has not been affected by cardinal changes: the classic temperature data storage is still used with the selection of the showcase area. Storefronts can be made using a wide variety of technologies that consume data from a single storage platform in a consistent model. This approach shows a good result and minimizes the work of adapting users to changes. It is important to highlight such an innovation as a "sandbox", or data marketplace, when the end-user can order some data set in the factory and independently work with it, implementing the concepts.

But even here new components appear, such as a data management system, or Data Governance: Data Catalog, Data Lineage. This also includes MDM and Data Quality systems, that is, all systems that manage data as a business entity. The subject of Data Governance is much broader than a data factory, it is already a separate area of knowledge, which includes organizational units, methodological regulations, software components.

And finally, the third conceptual layer, is aimed at the end-user of the data. This component is also becoming more complicated: various business services, data sandboxes, Data portals appear, where the user can supplement ready-made schemes.

As an example from the life around us, various applications are now being launched in large numbers by government agencies, banks, and telecommunications companies. Smart services help the end-user in various areas of life.

As for the technology stack that must be implemented before the launch of the Data Factory, Data Factory makes the most of the full potential of cloud technologies, virtualizing all components of the IT infrastructure. In this regard, companies should, first of all, ensure the uninterrupted and fast functioning of the clouds.

Summary

A data factory is a serious project that should involve not only IT experts but also business, both top management and all stakeholders. A lot of DevOps teams have emerged over the past few years. Why is this happening? The company wants to bring products to market quickly. This is impossible without clear interaction lines and consistency between different departments. Every team member needs to understand the role they play in the process of building the Data Factory and data lakes.